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ABSTRACT 

This paper reports the results of a qualitative 
evaluation of the Chapter 1 Program Improvement Workshop 
Series — Le^el II, funded by the Educational Testing Service, 
Technical Assistance Center (TAC) , for Reqion 3. Focus is on the role 
of commitment in influencing change. The methodological approach, 
advocated by M. Q. Patton Q986) , focuses on: (1; appropriateness, 
utility, practicality, credibility, and relevance to clients; and (2) 
"hidden agenda" issues or participants* gut-level reactions to 
programming. Hidden agenda can only be revealed through direct 
observation ancl face-to-face interviews. Factors identified as 
influencing commitment include information about the program, support 
at the state level, an emphasis on parent advocacy, and multiple 
roles played by TAC facilitators. Factors affecting willingness to 
change included commitment of new administrative personnel and the 
increased emphasis on the role of the Chapter 1 teacher. Finally, the 
use of exemplary and innovative programs was emphasized. The use of 
quali+'ative techniques of data collection allowed evaluators to play 
a much stronger role in formative evaluation. (TJH) 
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A key Issue facing any organizatiun Is how to facilitate i novative changes 
within the structure of existing programs, m her landmark stu.y of corporations 
that were known for their ability to encourage and implement innovative programs, 
Kanter (1983) developed a definition of innovation which can also be applied to 
the educational context: 

Innovation refers to the process of bringing any new, problem 
solving Idea Into use. ideas for reorganizing, cutting costs, 
putting in new budgeting systems [or evaluation systems], improving 
communication, or assembling products [or defining processes] In 
teams are also Innovations. Innovation Is the generation, 
acceptance, and implementation of new Ideas, processes, products, 
or services, it can thus occur in any part of a corporation (or 
school], and it can Involve creative use as well as original 
innovation. Application and implementation are centr ' to this 
definition: It Involves the capacity to change or adap And there 
can be many different kinds of innovation?, brought about by many 
different kinds of people: the corporate equivalent of entrepreneurs 
(Kanter, 1983, pp. 20-21). [Italics added by author of this paper] 

Based on this definition, two questions need to be considered: (1) What are the 
factors which get people to commit to innovative ideas; and, (2) What are the 
factors which affect their willingness to change and implement these ideas? The 
purpose of this paper is to suggest that questions of this nature can best be 
answered through a qualitative investigation, rather than just a comparison of 
program outcomes across sites. As Cronbach et al. (1980, p. 7) described it, 
"Merit lies not in the form of inquiry but In the relevance of information." 

This paper reports on the results of a qualitative evaluation of the 
Chapter 1 Program Improvement Workshop Series - Level II, which was funded by 
the Educational Testing Service, Technical Assistance Center - Region 3. The 
evaluation had two purposes: to address issues of concern to TAC in terms of 
whether the workshops were successful in accomplishing their goals and 
objectives, and to address the concerns of participants in terms of whether they 
were being provided with services of real value to them, and how new Ideas 
acquired from the workshops were being implemented. 

Background context 

As part of the U. S. Secretary of Education's initiative to improve the 
quality of Chapter 1 projects, a national p'-ogram was begun in 1984 to identify 
and recognize exemplary Chapter 1 projects, and to disseminate information about 
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these projects. This program was expanded in 1985 to include assistance in 
helping State and Local Educational Agencies (SEA, LEA) improve the quality of 
their own Chapter 1 programs. As a means of doing this, the ETS Technical 
Assistance Center (TAC) - Region 3 set up a series of program improvement 
workshops to oe conducted during a one and a half year cycle from November, 1987 
- March, 1989, in various southern states. Assistance from an external evaluator 
was requested in order to determine the effectiveness of the workshops among the 
participating SEA's and LEA's. TAC was interested in using an outside evaluator 
for several reasons: 1). they wanted to learn what was the effective level of 
intervention by TAC personnel with the oEA and LEA's involved in the project, 
particularly from the point of view of those receiving the services; 2). because 
these workshops represented both a new service and a new method of delivery 
(doing multi-visits in a collaborative format, assessing interaction as well as 
outcomes), they were interested in learning whether it was effective; and, 3). 
they hoped the results of this evaluation would lend credibility to this new 
approach within the field. 

One goal of the workshops was to help coordinators become more familiar 
with the process of program improvement, and to help them view improvement as 
an on-going activity, rather than as an issue which needed addressing only if 
the results of the summative evaluation (i.e, NCE gains) were negative. This 
goal reflects a fundamental shift in the nature of evaluation. In that 
evaluators become partners in facilitating change, instead of just identifying 
problem areas and recommending strategies for remediation. As one researcher put 
it, evaluators become "resources and facilitators whose job it is to develop the 
skills and confidence of all collective members of a project and to provide 
whatever support services are necessary for them to achieve what they want." 
(Kirkup, 1986, p. 76). 

A second goal was to provide districts with information and materials that 
would help guide them in the program improvement process. Before the sessions 
began, each district was mailed a packet of information which included the 
following documents: a description of the 13 attributes of exemplary Chapter 1 
programs, a selected bibliography organized around topics such as parent 
involvement, defining goals and objectives, etc., and on interpretation guide 
for Title 1 evaluation results. During each one of the sessions, participants 
were also provided with folders crammed full of handouts, forms, booklets, 
charts, etc. Although several participants noted that much of this was useful 
information for future reference, several commented that the sheer volume of 
information was somewhat overwhelming. As one participant put it, "I don't know 
when they think Tm going to have time to read all this stuff." 



Strategies for data collection 

The methodo 1 og i ca 1 approach f o 1 1 owed in th i s eva 1 uat i on is the one 
advocated by Patton (1986), who stated that evaluation methods should be judged 
on the basis of appropriateness, utility, practicality, credibility, and 
relevance to clients, and that these criteria are necessarily situational and 
context-bound. In other words, the methods should allow the evaluator to collect 
data which fits the clients' needs and interests, and not necessarily conform 
to an academically derived standard of methodological purity. "i designing a 
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qualitative vwaluation, evaluators need to consider further the goals of the 
evaluation from several perspectives. One perspective is, of course the 
questions that the client wishes to have addressed. To address the issues raised 
above, a set of survey questions was developed both from discussions with people 
in TAG, as well as from the information district people gave in informal 
interviews. 

However, whenever a new program is launched that advertises itself as 
"different" from usual practice, there are always two agendas to be considered. 
One agenda might be labeled the "open agenda," which is the set information 
presented as topics in the workshops, and which is evaluated using a form 
distributed after the workshops. In general, people tend to respond favorably 
to these forms, because they are given immediately afterwards before people have 
had a chance to process what they have learned, and to consider how these new 
ideas might be implemented. A review of the evaluation data collected by TAG 
using this format lends credence to this point, since the majority of responses 
were highly favorable. 

The second agendi might be labeled as the "hidden agenda," which is shaped 
by peoples' "gut level" reactions to what is being presented, as well as by 
factors influencing acceptance and implementation. This agenda is never captured 
through formal assessment; it is only visible over time through triangulation 
of data from observations, interviews, and document analysis. And even with 
extensive fieldwork, the evaluator never captures it all; undoubtedly, every 
evaluator (and perhaps every worKshop presenter) at some point wishes they could 
be a fly on the wall when the participants return to their home base and begin 
discussing the real issues of how these ideas will pla' in their district. 
However, a qualitative approach is useful and vitally neeued for studying this 
agenda, since witnout considering it, real changes in program improvement will 
not be accomplished. 

One example will suffice to illustrate the importance of looking at the 
"hidden agenda," although many more can be cited, in one of the workshop 
sessions observed, the topic presented dealt with the issue of coordination 
between the Ghapter 1 teacher and the regular classroom teacher. According to 
the federal regulations, both teachers should meet on a regular basis to ensure 
each child in the program is given appropriate instruction. One coordinator 
referred to this idea as "harmonious action." As the TAG presenter spoke, the 
audience nodded their heads in agreement, because everyone, of course, agrees 
with this idea, and everyone feels their school is handling this issue well. 

When the whole group broke out into breakout sessions to discuss this 
topic in more detail, a different picture began to emerge. The first 
disillusionment centered around the fact of establishing compliance with federal 
regulations. Several people noted that the state provided almost no guidelines 
as to what constitutes "optimal" coordination, and virtually no enforcement of 
whether school districts were in compliance or not. It was obvious that some 
teachers were irked at others from those schools they knew not to be in 
compliance, when they were spending a great deal of time in meetings. One 
elementary coordinator tartly remarked that "more kids' needs can be met if less 
meetings take place, and if the whole program was streamlined." 
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A second sore point was the Issue of teamwork and administrative support. 
Although teams are certainly strengthened by the inclusion of administrators, 
it's also the case that these same administrators can be become somewhat 
defensive when challenged by the teachers to give reasons for their decisions 
that Impede coordination. For example, several teachers mentioned that 
scheduling problems made it almost Impossible for Chapter 1 and regular 
classroom teachers to meet, and that in many cases. Chapter 1 teachers had no 
voice In scheduling at all. One woman said her Chapter 1 teachers went "up In 
arms" and confronted the principal to reserve a set block of time for meetings. 
Another coordinator commented upon the "hardened bureaucracy" present in her 
school and how difficult it was to promote any changes. Her hope was that 
attending these sessions would give her "ammunition" to push for changes based 
on changes in the federal regulations. 

Two points can be drawn from this example. One, In the flow and ebb of 
conversation, people's emotions and feelings were much more openly displayed 
than in the general meeting presentation. If this much feeling is revealed In 
the case where there was an observer taking notes, then one can only Imagine the 
conversations which occur at the school level, with no observer present. Some 
of these charged emotions did come through 1n separate interviews conducted 
privately away from the sessions. And two, none of this feeling was revoaled in 
the responses to the survey question asking districts to identify their weakest 
areas. Very few mentioned coordination, even though it is a sensitive issue -^or 
many schools. People are often reluctant to reveal sensitive topics when they 
have to write them down, but are much more willing to be quoted "off the 
record. " 

Data which reveal the hidden agenda are not collected through survey 
questions; they can only be captured successfully through direct observation and 
face-to-face Interviews. Although 1t is more time-consuming, data of this nature 
does reveal definitive patterns which can provide a substantive basis both for 
understanding the process of Implementing new innovations, and ensuring that 
changes made In the program will bo adopted by those for whom the program was 
designed. 

Factors influencing commitment to change 

The key factors identified as a result wh^ch affected the level of 
commitment were: 1). the Chapter 1 Program Improvement Series brought a renewed 
sense of commitment to the Chapter 1 program in both states among the 
participants Just from their hearing about new ideas and listening to other 
districts' experiences; 2). there was strong support at the state level. In both 
states, either the statewide or area Chapter 1 coordinators were strongly 
committed to the program, and their support gave people a reason to "buy Into 
it."; 3). a strong emphasis on parent advocacy, making parents feel more 
committed to the program by giving them a stronger voice in their children's 
education. One way proposed by several districts was a direct linkage between 
school and community resources; md, 4). the multiple roles played by the TAC 
facilitators. 

Each one of the factors identified above had Its own stories associated 
with It. Concerning the renewed sense of commitment, one supervisor reported 
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that her teachers just "blossomed" under staff development; she believed 
strongly that "teachers need to commit themselves to personal growth/* She also 
felt that this program had the state department of education's sanction to 
promote staff development in the schools. This aspect was important to this 
supervisor, since she noted her superintendent was "not real keen" on staff 
development, and these workshops gave her a greater sense of authority in 
pushing for more staff development in her school district. Another district 
Chapter 1 Director talked about the need to "hold hands and work on our problems 
together. " 

Another important factor was strong support at the state level. In both 
states, either the statewide or area Chapter 1 coordinators were strongly 
committed to the program, and their support gave people a reason to "buy into 
it." One reason for their support was that they saw these workshops as a more 
effective means of delivery of services; "the DOE could reach more people this 
way." The need for early intervention was also recognized; by using this 
service, the coordinators had a better handle on which districts really needed 
help. This was painfully obvious during the observation of a private conference 
with one "troubled" district. It became clear that the district team had no idea 
how to use the student trace, and that they really didn't understand the data 
(particularly the interpretation of test scores for both placement and 
evaluation) they were asked to bring, if the state had waited until this 
district sent in their yearly evaluation form which indicated no NCE gains (or 
worse, slippage from the year before), remediation would come too late 
(assuming, of course, that the DOE had the money and personnel to make on-site 
visits). But by having this district come to the workshops, their problems could 
be identified and discussed in a setting where help was more readily available. 

A third factor is the idea of parent advocacy, i.e., making parents feel 
more committed to the program by giving them a stronger voice in their 
children's education. One way proposed by several districts was a direct linkage 
between school and community resources. For example, schools could be considered 
as a resource for the whole community, and not Just be used on a 7 a.m.- 3 p.m. 
basis. This is a revisiting of an old idea proposed by Ivan Illich in the 
1960's, but perhaps the time has come for the educational establishment to 
consider more innovative uses ;f schools than just educational centers for 
children. Already in Florida there is serious discussion at the state level of 
making social services available in the schools, so there is a more effective 
coordination of care for abused and neglected children. 

Finally, a very important factor identified was the role of the TAC 
facilitator. To make these workshops effective, the facilitator hai to be a 
person who is open to new ideas, receptive to the feelings and mood of the 
participants, flexible in adjusting the agenda to accommodate last minute 
changes in the field, uses games, role playing, and natural language rather than 
"educationese," and possesses both a good sense of humor and the ability to see 
the absurdities of life in schools. This last characteristic may seem rather 
frivolous, but workshop presenters who received good evaluations on the public 
forms were often derided in private as being "boring and pompous." One school 
principal confided that he stopped attending the workshops because "they weren't 
telling me anything I didn't already know, and I don't need to drive 50 miles 
to be put to sleep." People clearly enjoyed themselves at these workshops, they 
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laughed and shared stories, and much of the success of these workshops can be 
traced back to the personal qualities and expertise of the TAC facilitators. 



Factors affecting th e willingness to chanoQ 

Another major factor is identifying the person (or persons) responsible 
for 'taking changes and seeing that they accept the program. An interesting 
pattern turned up, especially in Alabama, related to recent changes in 
administrative personnel. In almost every case where districts reported strong 
support for implementing changes suggested by the workshops, there was also a 
new person in a key decision making spot, either a new principal, or local 
Superintendent, or Chapter 1 director, in Alabama, this was true even at the 
state level, where both the statewide Chapter 1 Coordinator and Chapter 1 
evaluator had just assumed these positions. Both were strongly committed to 
program improvement and they candidly admitted that the persons previously 
occupying these positions had been opposed to using state money for program 
improvement. This finding parallels the one reported by Davis (1988) in his case 
study of three districts participating in the Program Improvement Workshop 
Series conducted by TAC - Region 4. in his words: 

Dramatic Improvement in association with this process 
is more likely to occur in schools or districts in which 
a participating leader is relatively new and receptive 
to taking on a more active role than in a district in 
which the administrator in either attending to quality 
issues already, or unwilling to risk breaking out of the 
traditional style of leadership-by-rule-enforcement 
(Davis, 1988, p. 6). 



The fact that a similar pattern occurred in two quite different regions 
lends credence to Davis's hyoothesis that the Chapter 1 Program Improvement 
Process "represents a potential political resource for a leader under a limited 
set of conditions" (1988, p. 6). One ^imitation is that even if one decision 
maker is new in the position, support often has to be gained from others who 
have held their positions for a long time, an example of what one participant 
called the "hardened bureaucracy." This was the case of one Chapter 1 
Coordinator from a very rural, poor district who was anxious to make changes, 
but who had to contend with a director on the verge of retirement. Although he 
attended all thrj sessions, he was unable to gain support to bring a whole team, 
thereby lessening his chances for making substantive improvements. 

A second important factor is an increased emphasis on the role of the 
Chapter 1 teacher. Several Chapter 1 teachers and supervisors reported that this 
process helped "validate" their efforts in the classroom. It also provided them 
with a forum through inclusion on teams to voice their concerns and to have them 
taken seriously. Several teachers commented that prior to their district's 
attendance at these workshops, they were not routinely included in the planning 
process, and were often treated as separate appendages in their schools. The 
fact that several districts noted that topics of grsat concern to them were 
recognition of teacher excellence and the related one of teacher burnout 
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suggests that the quality of the teaching force in Chapter 1 programs is central 
to establishing any real changes. 
Exemplary vs, Ipnov ative Programs 

In addition to helping LEA's improve their Chapter 1 programs, a second 
objective for TAC was to help SEA's identify exemplary Chapter 1 projects within 
the state. Part of the information mailed to districts participating in the 
workshops included a description of the 13 attiibutes listed by the federal 
government as characteristics of exemplary programs. While these attributes are 
certainly behaviors that would characterize excellent programs (in fact, they 
can be considered exemplary practices for any school program, not just Chapter 
1), the question can be raised as to whether these attributes represent truly 
"Innovative" Ideas in the sense proposed by Kanter (1983). Here the issue of 
context becomes very Important, because many well funded school districts who 
can afford to employ their own curriculum development coordinators and research 
and evaluation personnel already engage in many of these practices. For these 
districts, the services provided by TAC in the workshops do not break "new" 
ground; they merely reiterate what the districts are already doing. And even in 
the case of more rural, less well-funded districts, where many of the workshop 
topics FAG offered were "new", people were vaguely aware that other alternatives 
to standard practices were available, or more importantly, they wished there 
were alternatives because they were dissatisfied with the present state of 
affairs. 

An example based on a long interview conducted in a rural, poor, 
predominately minority district illustrates this point. After the routine 
questions had been asked and answered, and the district team (all minority 
members) began to relax with the evaluator (a white female), the whole tenor of 
the Interview shifted. The Chapter 1 director began to speak of his dreams for 
the students in the district, and of his desire to "open up their horizons," As 
he put it, 'we've got to do right by our children, because they will be the next 
generation, and what will this country's future be like if they can't cope?" 
This same director was also actively engaged in promoting the use of computers 
for instruction in his Chapter 1 program, because he strongly felt that computer 
knowledge would be the key to his students' future. Another team member shared 
her frustration over her grandson's struggle to learn to read, and mused, "there 
has to be a better way to teach him. He's just not getting these worksheets at 
all." This same team is the group that went to Toronto for the IRA conference, 
and came back full of enthusiasm for implementing some of the ideas based on the 
whole language approach to reading instruction. What became clear as the 
Interview progressed is that these people shared a vision of what education 
could be like for their students, and that they looked to outside sources like 
the TAC workshops to give them support In getting their ideas Implemented. While 
this particular district was one of the more enthusiastic supporters of the 
workshops In that they perceived it "helped open up new vistas", even they 
looked for something more. As the director said at the end, "What we lack are 
ideas, not the materials or willingness to change." 

Raising this issue had two implications for TAC personnel. One, the 
interview data revealed that a more careful needs assessment needs to be done 
at the district level to determine what topics are most useful for that 
particular district, or even region. While TAC cannot obviously fine tune the 
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workshops to meet every LEA's unique needs, a better match can be made to ensure 
that the topics presented do not already duplicate the expertise within the 
district. And two, if TAC is going to assume the task of providing what Patton 
(1986) called more "front-end" assistance, then they need to consider whether 
that role should be limited to simply helping programs become more adept at 
utilizing existing practices of excellent schools, or whether they should take 
a more proactive stance in disseminating information about ideas that are on the 
"cutting edge" of educational research in teaching and learning. 

Conclusion 

Any qualitative evaluation that is done well raises new questions in 
addition to addressing the ones posed by the client. And that is because many 
issues and concerns cannot be anticipated in advance, but are discovered only 
by entry into the field. One question is focused on the evaluator's role in 
collecting and analyzing the data. The results of this evaluation suggest that 
evaluators can play a much stronger role in formative evaluation, especially if 
they are sensitive to information that uncovers multiple realities in program 
implementation. As one evaluator of a Chapter 1 program described it: 

I see evaluators being given a role in program 
development. In this role, we are working side by side 
with the program personnel in an effort to design and 
implement a program that works - and works well (Rallis, 
1988, p. 29). 



A second point to be made is that qualitative methods are useful for 
uncovering peoples' real reasons for becoming committed to making what in many 
cases were innovative changes in their program. Unless program administrators 
have this information, evaluation reports will continue to gather dust on the 
shelves in bureaucratic offices, and the problems associated with program 
implementation will not be addressed. 
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